Questions

What do these have in common?

Techniques to reduce sensitivity bias

• Honesty appeals

• Confidentiality protocols

• Randomized response

• Network scale-up

• Endorsement experiments

• List experiments

Techniques to reduce sensitivity bias

• Honesty appeals

• Confidentiality protocols

• Randomized response

• Network scale-up

• Endorsement experiments

• List experiments

Example

List experiment

Here is a list of things that some people have done.

List experiment

Please listen to them and then tell me HOW MANY of them you have done in the past two years.

List experiment

Do not tell me which ones. Just tell me HOW MANY:

List experiment

Do not tell me which ones. Just tell me HOW MANY:

Treatment group

1. Discussed politics with family or friends
2. Cast a ballot for governor Phil Bryant
3. Paid dues to a union
4. Given money to a Tea Party candidate

List experiment

Do not tell me which ones. Just tell me HOW MANY:

Treatment group

1. Discussed politics with family or friends
2. Cast a ballot for governor Phil Bryant
3. Paid dues to a union
4. Given money to a Tea Party candidate
5. Voted “YES” on the Personhood Initiative

Prevalence rate

\[ \text{Proportion(Voted yes)} =\\ \text{Mean(List with sensitive item)} -\\ \text{Mean(List without sensitive item)} \]

library(estimatr)
difference_in_means(count ~ sensitive, data = df)

Compare with direct question

Did you vote YES or NO on the Personhood Initiative, which appeared on the November 2011 Mississippi General Election Ballot?

t.test(yes ~ 1, data = df)

Validation

Validation

Validation

Validation

Sensitive item

Organization X (advocating for immigration reduction and measures against undocumented immigration)

Three prevalence estimators

difference_in_means(count ~ sensitive, subset = list == "A", data = df)

difference_in_means(count ~ sensitive, subset = list == "B", data = df )

lm_robust(count ~ sensitive + list, clusters = id, data = df)

DLE yields more precise estimates

DLE yields more precise estimates

But variance reduction is not free

But variance reduction is not free

DLE variants

DLE variants

• Fixed-fixed is not an admissible design

DLE variants

• Fixed-fixed is not an admissible design
• Randomized-fixed keeps sensitive item in second list

DLE variants

• Fixed-fixed is not an admissible design
• Randomized-fixed keeps sensitive item in second list

DLE variants

• Fixed-fixed is not an admissible design
• Randomized-fixed keeps sensitive item in second list
• Fixed-randomized and randomized-randomized shuffle sensitive item order

Carryover design effects

Toy example

Toy example

Toy example

Toy example

Toy example

Toy example

Toy example

Toy example

Why does this happen?

Statistical tests

Statistical tests

1. Difference-in-differences

2. Signed-rank test

Difference-in-differences test

\[ \hat{\tau}_1 = \text{Mean}(\text{First list}_t) - \text{Mean}(\text{First list}_c) \]

\[ \hat{\tau}_2 = \text{Mean}(\text{Second list}_t) - \text{Mean}(\text{Second list}_c) \]

# For fixed-randomized design
lm_robust(count ~ sensitive*first, clusters = id, data = df)

Application to Alvarez et al (2019)

Application to Alvarez et al (2019)

Application to Alvarez et al (2019)

Criminal governance tools in Uruguay

• Facebook sample of Montevideo residents (N = 2688)

• Four criminal governance strategies

Criminal governance tools in Uruguay

• Facebook sample of Montevideo residents (N = 2688)

• Four criminal governance strategies

DLE with placebo item

Things people have experienced in the last six months:

DLE with placebo item

Things people have experienced in the last six months:

DLE with placebo item

Things people have experienced in the last six months:

Prevalence estimates

Prevalence estimates

Prevalence estimates

What went wrong?

Applying difference-in-differences test

Applying difference-in-differences test

So we can move from this

To estimate bounds

To estimate bounds

To estimate bounds

Conclusion

Summary

Conclusion

Lessons

Appendix

• Alvarez et al (2019) details
• Montevideo survey responses
• Signed rank test
• DLE cost simulations
• Performance simulations
• Placebo tests
• Network scale-up

Alvarez et al (2019) details

Mean supported organizations

Control list distributions

Montevideo survey

Montevideo survey

Montevideo survey

Montevideo survey

Stephenson’s signed rank test

\[ \widetilde{T} = \sum_{i=1}^N \text{sgn} \{(z_i - (1-z_i)) (Y_{i1} - Y_{i2})\} \times \tilde{q}_i \]

\[ \tilde{q}_i = {q_i-1 \choose m-1} \text{ for } q_i \geq m \]

\[ \tilde{q}_i = 0 \text{ for } q_i < m \]

\[ \text{with } 1 \leq m \leq N \]

Applied to Alvarez et al (2019)

DLEs are costly to implement

Test performance

Signed rank performance

Placebo tests

Network scale-up method

Network scale-up method

How many X do you know, who also know you,

Network scale-up method

How many X do you know, who also know you, with whom you have interacted in the last year

Network scale-up method

How many X do you know, who also know you, with whom you have interacted in the last year in person, by phone, or any other channel?

Network scale-up method

How many X do you know…

NSUM results